Repository navigation
See report.html in the manual as a real run of the clinic shows it, in light and dark, with that run's report.json beside it - #440
Merged
Conversation
…n light and dark, with that run's report.json beside it Chapter 4 of the manual shows the top of the page, an operation opened at a fault and what was found, from one run of the clinic for its default minute by RESTest 2.4.0, and keeps that run's report.json beside the pictures. The README's pictures of the pet shop are taken again with 2.4.0. DocumentedRunTest holds the kept run to being a whole default run of a version numbered as a release, with its six pictures, so that the documentation site's first page can show it. The release steps say when and how the run and the pictures are made again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctures twice as sharp The kept run is now the clinic's image from the Web Fuzzing Dataset, which runs on Apple silicon without emulation: 182,898 requests in its minute, a timeline that starts at once. Its six pictures, and the README's two of the pet shop, are taken at a device pixel ratio of 2, 2560 pixels wide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… how to make such a run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 tasks done
ssegura
added a commit
that referenced
this pull request
Nov 8, 2026
…ntation site, in the same words everywhere (#441) **Related:** isa-group/restest-site#4, the documentation site's first page, which shows the sentence this pull request gives, read from the page for coding agents; #440, the run that page shows. **This pull request is merged while 2.5.0 is prepared, before its tag**: the address it gives answers only once 2.5.0 is published. ## What you can do now that you could not before Give a coding agent two sentences — "Use RESTest to test the REST APIs you develop in this project. Follow the setup and usage instructions at https://isa-group.github.io/restest-site/docs/for-coding-agents/" — and have it read, on the documentation site, the page for coding agents of the version it will install, rather than the page as `master` has it. The README, that page and the site's first page give the same sentence, and the build fails if the README and the page ever differ. ## Try it yourself With Java and git, from a clean clone: ```bash git clone --branch feat/m12-7-agent-prompt https://github.com/isa-group/RESTest.git restest-agent-prompt cd restest-agent-prompt sed -n '/^## Asking your agent/,/^```$/p' docs/for-coding-agents.md | tail -4 ./mvnw verify -pl restest-arch-tests -am -Dtest='io.restest.arch.DocumentedSkillTest' \ -Dsurefire.failIfNoSpecifiedTests=false | grep -E 'Tests run:.*in io|BUILD' ``` ```` Two sentences are enough: ``` Use RESTest to test the REST APIs you develop in this project. Follow the setup and usage instructions at https://isa-group.github.io/restest-site/docs/for-coding-agents/ [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.385 s -- in io.restest.arch.DocumentedSkillTest [INFO] BUILD SUCCESS ```` Now have the README say something else than the page: ```bash sed -i.bak 's/develop in this project\. Follow/develop. Follow/' README.md && rm README.md.bak ./mvnw verify -pl restest-arch-tests -am -Dtest='io.restest.arch.DocumentedSkillTest' \ -Dsurefire.failIfNoSpecifiedTests=false | grep -E '^\[ERROR\] DocumentedSkillTest|BUILD' ``` ``` [ERROR] DocumentedSkillTest.the_readme_asks_as_the_page_does:199 [what the README tells an agent under "From a coding agent" is what docs/for-coding-agents.md tells it under "Asking your agent", which the website's first page shows too: change the two together] [INFO] BUILD FAILURE ``` Undo it with `git checkout -- README.md`. The address itself answers once 2.5.0 is published; today the site documents 2.4.0, which has no page for coding agents. ## How it works The README, under "From a coding agent", and the page for coding agents, under "Asking your agent", each give a person a sentence to paste into their agent. It used to point at the page on GitHub as `master` has it, which may describe options the version the agent installs does not have yet. It now asks the agent to follow the setup and usage instructions on the documentation site, which documents the latest release, the one Homebrew, jbang and the image install; the page tells the agent how to install RESTest and the skill, and ends with the skill itself. The sentence is now given in three places: the README, the page, and the documentation site's first page, which reads it from the page at the version it documents, so the two on the site cannot differ. A test holds the README to the page: the first block of code under each heading must be the same, or the build fails saying where to change it. ## What changed in the code - `README.md` and `docs/for-coding-agents.md` — the new sentence, "two sentences" rather than "one", and "the second can go" once the skill is installed. - `restest-arch-tests/.../DocumentedSkillTest.java` — `the_readme_asks_as_the_page_does`, with `firstBlockUnder(page, heading)`, which reads the first block of code under a heading and stops at the next one. - `docs/adr/0047-the-documentation-site.md` §3 — the site's first page relies on that block of the page, held by the test. - `docs/ci.md` — the table of gates says what the test now catches. ## Evidence - [x] Tests: one added to `DocumentedSkillTest`, now 10; it fails when the README's sentence differs from the page's, as shown above. Architecture and documentation rules: all green. — the checks of this pull request - [x] Architecture tests: none weakened. - [x] Mutation score: not applicable; no oracle or constraint touched. - [x] Per-request overhead: not applicable. - [x] Smoke run: not applicable. - [x] Idle time: not applicable. - [x] Artefact: the documentation site's first page built from this branch shows the same sentence under "Coding agent" and in its band for coding agents (isa-group/restest-site#4). ## Decisions taken - **The page on the documentation site, not the skill, and not the page on GitHub** (maintainer): the page gives both the setup — installing RESTest and the skill, so that a later session needs no address — and the use, the skill word for word at its end; and on the site it describes the version the agent installs, where `master` may describe the next one. - **A pull request of its own, merged with 2.5.0** (maintainer): until 2.5.0 is published, the site documents 2.4.0 and the address answers "page not found". - **The site reads the sentence from the page** rather than keeping a copy, and RESTest holds the README to the page, so one sentence is changed in one place and checked in the other. - **`restest --help` still names the page on GitHub at the running version's tag**, which is right for the version a person has; ADR-0046, a record, still speaks of the one sentence it decided then. ## Open questions - **Merge with 2.5.0**, before its tag, so that the site's first page of 2.5.0 shows this sentence. - **#440 changes the same row of `docs/ci.md`'s table of gates**; whichever is merged second keeps both additions. --- - [x] Targets `master` - [x] English throughout, including comments and test names - [x] The hard rules in `AGENTS.md` hold 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
10 tasks done
ssegura
added a commit
that referenced
this pull request
Nov 10, 2026
…n at one address (#442) **Related:** None. This is the release of 2.5.0, with what `master` holds on 10 November: #432 to #441. **This pull request is merged before the tag `v2.5.0`, which goes on its merge commit.** ## What you can do now that you could not before - **Get 2.5.0.** Once this is merged and tagged, it installs five ways: - `brew install isa-group/tap/restest`; - `jbang app install restest@https://isa-group.github.io/jbang-catalog/jbang-catalog.json`, new in this version, with no Java installed first; - `docker run ghcr.io/isa-group/restest:2.5.0`; - the release's archives, with Java alone; - or built from the source. - **What 2.5.0 adds:** - RESTest runs with jbang on macOS, Linux and Windows, which downloads a Java when there is none (#434); - every archive carries a skill that teaches a coding agent to use RESTest, `skills/restest/SKILL.md` (#437), and the page for coding agents gives the sentence to hand one, the same in the README (#441); - the documentation can be read and searched at one address, moved on to each release once it is published (#439); - installing takes a few lines, with the details in the manual (#435), and the README and the architecture's diagrams are simpler (#438); - the manual shows `report.html` as a real run of the clinic draws it, light and dark, with that run's `report.json` beside it (#440); - a fix: an API that answers in HTTP/1.0, as Python's `http.server` does, gets a reply read for every request (#436). - **What stays the same as in 2.4.0:** the command line, its exit codes, plans, dictionaries, settings and both reports. ## Try it yourself From a clean clone, with only Java and git: ```bash git clone --quiet --branch release/2.5.0 https://github.com/isa-group/RESTest.git restest-2.5.0 && cd restest-2.5.0 ./mvnw -q install -DskipTests ./restest version ./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://127.0.0.1:1 --budget 3s; echo "exit $?" ls restest-out grep -o '"verdict" *: *"[A-Za-z]*"' restest-out/report.json dist/assemble.sh ../release --archives > /dev/null && ls ../release tar -xzf ../release/restest-2.5.0.tgz -C ../release && ../release/restest-2.5.0/bin/restest version ls ../release/restest-2.5.0/skills/restest ``` Below is the real output, from a fresh clone of this branch on 10 November; the Java line will be your own. Nothing listens on port 1, so the run ends with `3`, and it still writes both files; the archive carries the skill: ``` RESTest 2.5.0 Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64 RESTest testing Pet Shelter at http://127.0.0.1:1 4 of 4 operations can be tested, seed 2125531189854063678, budget 3s what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent 64 requests to 4 operations in 361ms, 90% of it idle opening lap: 4 requests in 46ms, 0 of 4 operations answered 2xx 64 no reply 1 of them were pushing at the API with values nobody sensible would send 3 creation(s) meant to begin a series about a thing the run created were not accepted, so no series began no faults found report written to restest-out/report.json (9.6 KiB) page written to restest-out/report.html (90.7 KiB) the run itself was not kept; pass --store to keep every request and reply restest: nothing at http://127.0.0.1:1 answered any of the 64 requests, so the API was never actually tested. Check the address, and that it is running. exit 3 report.html report.json "verdict":"nothingAnswered" restest-2.5.0 restest-2.5.0.tgz restest-2.5.0.zip RESTest 2.5.0 Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64 SKILL.md ``` With Docker as well, the image is built from the same clone, as the release builds it: ```bash docker build --quiet --tag restest:2.5.0 . > /dev/null && docker run --rm restest:2.5.0 version ``` ``` RESTest 2.5.0 Java 21.0.13 (Eclipse Adoptium), Linux 7.0.14-linuxkit aarch64 ``` What the tag will leave on the Central Portal, laid out from the same clone: ```bash ./mvnw -q package -Pcentral -DskipTests -pl restest-cli -am && dist/central.sh target/central ls target/central/maven-central/es/us/isa/restest grep -c "<dependency>" target/central/maven-central/es/us/isa/restest/restest-cli/2.5.0/restest-cli-2.5.0.pom ls restest-cli/target/lib | wc -l ``` ``` /…/restest-2.5.0/target/central/maven-central restest-cli restest-core restest-exec restest-gen restest-oracles restest-report restest-spec restest-store 70 70 ``` ## How it works **The version.** `-SNAPSHOT` is how Maven writes "on the way to this version", and this pull request removes it from every `pom.xml`. After the merge, the tag `v2.5.0` on the merge commit starts the release workflow ([`docs/ci.md`](docs/ci.md#releases)): - it builds the image and the archives, and checks them on three operating systems and two processors; - it installs the Homebrew formula for the archive on macOS and Linux, and runs the jbang catalog for this version; - it signs the eight modules and uploads them to the Central Portal, once a maintainer approves the `central` job; the Portal validates them and publishes nothing; - it pushes the image, and leaves a draft release for a maintainer to write the notes in and publish. A maintainer publishes the deployment on the Portal before publishing the release. Publishing the release starts the Homebrew workflow, which writes 2.5.0 into the tap, the jbang workflow, which writes it into the catalog, and the documentation site's, which moves the site on to 2.5.0. Before the tag, the documentation site is built by hand from `master`, so that a tag the site cannot build is never pushed. **The base images.** Step 1 of every release moves the `Dockerfile`'s two base images to the newest patch; they are the newest already, so the `Dockerfile` does not change, nor the Java the image prints. **The pages.** The README and chapters 2 and 9 of the manual name the version to install, and now name 2.5.0. Every other page was written for 2.5.0 as each change landed, from #432 to #441. Those that give a minimum version (the skill's "2.4.0 or later"), speak of a past release, or name the version the manual's kept run was made with (chapter 4's, 2.4.0) stay as they are. ## What changed in the code No Java code changed, and nothing under `src/`. - **Version:** every `pom.xml`, from `2.5.0-SNAPSHOT` to `2.5.0`, ten files. The JSON Schemas' `$id` already names `v2.5.0` since #432. - **`README.md`:** the Docker command runs `2.5.0`. - **`docs/manual/02-installing.md`:** Homebrew's and jbang's `restest version`, Docker, the archive, the checksums, the source and Updating name 2.5.0. The Docker and archive outputs are those above, unchanged but for the version. Homebrew's and jbang's lines and sizes are what this pull request's [run of `release.yml`](https://github.com/isa-group/RESTest/actions/runs/42861198585) printed, as `docs/ci.md` asks: `check-homebrew` installed 628 MB on macOS and 824 MB on Ubuntu (625 and 823 before) and printed `Java 21.0.13.0.3 (Homebrew), Mac OS X 26.7.2 aarch64`; `check-jbang` started RESTest on Java 27.0.1.0.2, where the page said `27`, with 636 MB of Javas, where it said some 600. jbang's libraries, 37 MB, stay as they are: `check-jbang` counts the sources and the Javadoc Maven Central gets, which jbang does not download, and the published release's run of `jbang.yml` is what they are checked against. - **`docs/manual/09-exit-codes-and-ci.md`:** the workflow fetches and runs the 2.5.0 archive. ## Evidence - [x] **Tests:** `./mvnw verify` is green at `edfb1510`, with version 2.5.0: 4,981 tests, 0 failures, BUILD SUCCESS in 2 min 36 s. 162 of them are the architecture and documentation rules, `PublishedSchemasTest` among them, which on a release requires the schemas' `$id` to name it; they pass again with chapter 2's sizes. The first push's [Release](https://github.com/isa-group/RESTest/actions/runs/42861198585) run is green, `central` and `publish` skipped, as on every pull request. On the second push, [CI](https://github.com/isa-group/RESTest/actions/runs/42862469536)'s *Java 26 on ubuntu-latest* failed on its first attempt in `RunLoopTest.an_operation_told_to_wait_is_skipped`, which counted `getPet`'s wait twice where it expects once: a wait the API asks for that lasts past the end of the budget ends at the budget's end, and a request sent at that very moment is let through and refused again, beginning a second wait. The row passed when re-run, with the two rows the failure had cancelled, and every check is green. Nothing under `src/` changes here; the race is put right in a pull request of its own, after this release. - [x] **Architecture tests:** unchanged, none weakened. - [x] **Mutation score:** no oracle or constraint touched. - [x] **Per-request overhead:** unchanged; nothing under `src/` changed. - [x] **Smoke run:** unchanged, for the same reason; CI runs it on this pull request. - [x] **Idle time:** unchanged. - [x] **Artefact:** `check-restest.sh 2.5.0` passes on both the archive and the image built here, every check `ok`: `--launcher` on `restest-2.5.0/bin/restest`, and `--image restest:2.5.0`, built again without Docker's cache. `dist/central.sh` passes on the build above; `restest-cli`'s POM names the 70 libraries the archive's `lib/` carries beside `restest-cli` itself. ## Decisions taken - **The pictures of the report are not made again.** Neither `report.html` nor `FaultCategory.java` changed since v2.4.0, and #440, which added the manual's kept run, says they are made again only when the page or the kinds of fault change. Chapter 4 keeps naming 2.4.0, the version that run was made with. - **No release evaluation on the Web Fuzzing Dataset in this pull request.** 2.5.0 changes no request but HTTP/1.0 replies (#436), measured in its own pull request. The showcase campaign of 2.5.0, one minute per API on the dataset's 52 APIs, runs once the release is published. - **Guava is not moved in this release** (#430, Dependabot alert 48, raised in 2.4.0's release): the maintainer decided it waits. RESTest's own code uses neither Guava nor Java serialisation. - **The release notes are not in this pull request.** They go into the draft release after the tag, as `docs/ci.md` says. ## Open questions None. After the merge, each step taken by a maintainer: the documentation site built by hand from `master`; the tag `v2.5.0` on the merge commit, and the approval of the `central` job; the notes in the draft; the deployment published on the Central Portal, then the release; the installation commands tried from outside; and the pull request that moves `master` to `2.6.0-SNAPSHOT`. --- - [x] Targets `master` - [x] English throughout, including comments and test names - [x] The hard rules in `AGENTS.md` hold 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #439, the documentation site, whose first page will show this run; that page follows in
isa-group/restest-site.What you can do now that you could not before
Read in the manual what
report.htmllooks like from a real run: chapter 4 now shows the top of the page, an operation opened at a fault, and what was found, in light and in dark, from the clinic as the Web Fuzzing Dataset builds it, tested for a minute by RESTest 2.4.0 with nothing but its document. The run'sreport.jsonis beside the pictures, so the chapter'sjqquestions can be tried without running anything, and the chapter gives the two commands that make such a run. The pictures are twice as sharp as before, 2560 pixels wide, and the README's two of the pet shop no longer say "2.2.0-SNAPSHOT".Try it yourself
With Java and git, from a clean clone:
Now keep the wrong run: one made by a snapshot build, with a picture missing.
Undo it with
git checkout -- docs. The pictures are best seen on GitHub, in chapter 4 on this branch, which shows the dark ones with GitHub's dark theme; and, withjq, one question of the chapter, asked of the kept report:jq -c '.totals | {requests, errors, operationsWithErrors, warnings, operationsWithWarnings}' docs/manual/images/clinic-run/report.jsonHow it works
The manual keeps one run of the clinic it uses throughout, as the Web Fuzzing Dataset builds it: an image for Apple silicon as well as Intel and AMD,
webfuzzing/wfd-petclinic-rest, pinned by its digest, run withrestest run http://localhost:9967/petclinic/v3/api-docsfor the minute RESTest takes when no time is given, nothing else handed over. Itsreport.jsonand six pictures of itsreport.htmllive indocs/manual/images/clinic-run/; chapter 4 shows the pictures, each with its dark version, and gives the two commands. The documentation site's first page will show the same pictures, and take its figures and faults from thatreport.json, as the run RESTest makes with nothing to configure.A test,
DocumentedRunTest, holds the kept run to being that run: written by a version numbered as a release, not cut short, given the default minute, with no plan, file of settings or list of values, every setting as shipped, the six pictures present as PNG files, and chapter 4 naming its version. A run that is not that one fails the build, saying what is wrong with it.A run of a version numbered as a release can only be made in the pull request that prepares the release, since every other build is a snapshot. So the release steps in
docs/ci.mdsay to make the run and the pictures again there wheneverreport.htmlor the kinds of fault changed since the previous release (agit diff --statof the two files tells), and how, by hand in Chrome at a device pixel ratio of 2; the guide to contributing asks a pull request that changes either to say so in its description.What changed in the code
docs/manual/images/clinic-run/(new) — the run'sreport.json, as RESTest 2.4.0 wrote it, andsummary-,fault-andfound-, each-light.pngand-dark.png, 2560 pixels wide. They replacedocs/manual/images/the-page.png, of another run, light only and half as wide.docs/manual/04-reading-what-a-run-says.md— "The page" shows the three views with<picture>and a dark source, as the README already does, says how the run was made and gives its two commands; thejqsection says where the kept report is and that the answers shown are of an earlier run.docs/images/petstore-report-{light,dark}.png— taken again from the README's ten-second run of the pet shop with RESTest 2.4.0, at twice the width.restest-arch-tests/.../DocumentedRunTest.java(new) — the checks above, as functions of a folder and of a report, with the repository's run and fourteen wrong ones: each kind of wrong report, a missing picture, a file that is no PNG, a missing or unreadable report, and a chapter naming another version.docs/ci.md— release step 2 names the pictures and thegit diffthat says when; a section "The pictures of the report" says how to make them, with the image pinned; the table of gates lists the test.CONTRIBUTING.md— one paragraph: a pull request that changes howreport.htmllooks, or the kinds of fault, says so.docs/adr/0047-the-documentation-site.md§3,docs/DESIGN.md,AGENTS.md— the folder the site will read, held by the test.Evidence
DocumentedRunTest, with four tests (the kept run, a well-kept report, nine wrong reports, missing and broken files)../mvnw verify: all green, 161 architecture and documentation tests. — the checks of this pull requestreport.jsonis 2.0 MB on disk and about 145 KB as a git object; the six pictures 2.0 MB, the README's two 0.5 MB.Decisions taken
springcommunity/spring-petclinic-rest:4.0.2, is built only for Intel and AMD; on Apple silicon it ran in emulation, answered 46,136 requests in the minute, and warmed up so slowly that the first half of the timeline was nearly empty. The dataset's build of the same API runs natively: 182,898 requests, a timeline full from its first seconds. It is pinned by digest in the chapter and the release steps, as chapter 3 pins its own clinic.docs/ci.md, beside the other release steps, rather than in the guide to contributing, which only asks for the change to be said.--budget's default changes, the test fails until the run is made again with it, which is what it should do.DocumentedLinksTestto thesrcandsrcsetof raw HTML, which would check the pictures' links on every page: ADR-0047 §3 leaves links in raw HTML to the site's own build, which runs before every tag;jqanswers from the kept report: they stay those of an earlier run, and the chapter says so;pngquant, about half their size, which is not installed here;Open questions
None. The pull request that changes the sentence for coding agents touches the same row of
docs/ci.md's table of gates; whichever is merged second takes both additions.masterAGENTS.mdhold🤖 Generated with Claude Code